Transparently enforcing policies in hadoop-style processing infrastructures

ABSTRACT

Method, system, and computer program product to facilitate selection of data nodes configured to satisfy a set of requirements for processing client data in a distributed computing environment by providing, for each data node of a plurality of data nodes in the distributed computing environment, nodal data describing the respective data node of the plurality of data nodes, receiving a request to process the client data, the client data being identified in the request, retrieving the set of requirements for processing the client data, and analyzing the retrieved data policy and the nodal data describing at least one of the data nodes, to select a first data node of the plurality of data nodes as a delegation target, the first data node selected based on having a higher suitability level for satisfying the set of requirements than a second data node of the plurality of data nodes.

BACKGROUND

Embodiments disclosed herein relate to computer software whichtransparently enforces policies in distributed processinginfrastructures.

Today, for large-scale data processing systems supporting cloudcomputing use cases, distributed file systems such as Hadoop have beenproposed. While Hadoop-based systems provide distributed file systemcapabilities with a decentralized architecture allowing superior levelsof business resiliency even if entire racks of server and storagesystems become unavailable due to network connectivity loss, hardwarefailure, or a disaster, Hadoop solutions (and distributed computingsolutions in general) are unable to support policy-driven service levelagreements for external customers in a transparent manner.

Hadoop is a software framework that supports data-intensive distributedapplications. Hadoop enables applications to work with thousands ofcomputational independent computers and petabytes of data. The Hadoopdistributed file system (HDFS) is a distributed, scalable, and portablefilesystem for the Hadoop framework. A large Hadoop cluster may includea dedicated name node which hosts a filesystem index to manage the HDFS,as well as multiple data nodes which may store data and performoperations on the data. Today, Hadoop and other distributed processinginfrastructures assume that all data nodes in their systems have thesame characteristics. (Hadoop is a trademark of the Apache SoftwareFoundation.)

SUMMARY

Embodiments disclosed herein provide a computer-implemented method, asystem, and a computer program product to facilitate selection of datanodes configured to satisfy a set of requirements for processing clientdata in a distributed computing environment by providing, for each datanode of a plurality of data nodes in the distributed computingenvironment, nodal data describing the respective data node of theplurality of data nodes, receiving a request to process the client data,the client data being identified in the request, retrieving the set ofrequirements for processing the client data identified in the request,and analyzing the retrieved data policy and the nodal data describing atleast one of the data nodes, to select a first data node of theplurality of data nodes as a delegation target, the first data nodebeing selected based on having a higher suitability level for satisfyingthe set of requirements than a second data node of the plurality of datanodes.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description ofembodiments of the invention, briefly summarized above, may be had byreference to the appended drawings.

It is to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIGS. 1A-1B are block diagrams illustrating a schematic to transparentlyenforce policies in distributed processing infrastructures, according toone embodiment disclosed herein.

FIG. 2 is a block diagram illustrating a system for transparentlyenforcing policies in distributed processing infrastructures.

FIG. 3 is a flow chart illustrating a method to transparently enforcepolicies in distributed processing infrastructures, according to oneembodiment disclosed herein.

FIG. 4 is a flow chart illustrating a method for identifying data nodessuitable to accept a data processing workload, according to oneembodiment disclosed herein.

FIG. 5 depicts a cloud computing environment, according to oneembodiment disclosed herein.

FIG. 6 depicts abstraction model layers, according to one embodimentdisclosed herein.

DETAILED DESCRIPTION

Embodiments generally provide techniques to facilitate selection of datanodes configured to satisfy a set of requirements for processing clientdata in a distributed computing environment. Embodiments may provide,for each data node of a plurality of data nodes in the distributedcomputing environment, nodal data describing the respective data node ofthe plurality of data nodes. A request to process the client data may bereceived, the client data being identified in the request. Additionally,embodiments may retrieve the set of requirements for processing theclient data identified in the request, and may analyze the retrieveddata policy and the nodal data describing at least one of the datanodes, to select a first data node of the plurality of data nodes as adelegation target. Here, the first data node could be selected based onhaving a higher suitability level for satisfying the set of requirementsthan a second data node of the plurality of data nodes.

Although embodiments disclosed herein use the Hadoop data storageframework as a representative example, embodiments are not limited tothe Hadoop framework. Rather, it is broadly contemplated thatembodiments can be extended all types of distributed file systems, knownor unknown. Embodiments disclosed herein provide a dynamic cloudinfrastructure by improving capabilities of a name node to assemble“sub-clouds” from the total set of all data nodes with differentcharacteristics expressed in metadata based on need. Furthermore,embodiments provide enhanced metadata describing the data nodes, whichare not considered equal in light of privacy policies, securitypolicies, retention policies, and disaster recovery policies. Forexample, embodiments provide different types of data nodes supportingdifferent functional and non-functional policies for the end consumer.In such an embodiment, only the name node may have the metadatanecessary to distinguish between the different data node types, and maydynamically select which data nodes are used for a particular task.Additionally, the name nodes could utilize technical metadata describingdifferent characteristics of the data nodes, such as computing power,storage size, network bandwidth, location, etc., in order to select themost appropriate data nodes to complete a computing task. As usedherein, “nodal data” includes the metadata which describes thecharacteristics of each data node in the infrastructure.

In the following, reference is made to embodiments of the invention.However, it should be understood that the invention is not limited tospecific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theinvention. Furthermore, although embodiments of the invention mayachieve advantages over other possible solutions and/or over the priorart, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the invention. Thus, the followingaspects, features, embodiments and advantages are merely illustrativeand are not considered elements or limitations of the appended claimsexcept where explicitly recited in a claim(s). Likewise, reference to“the invention” shall not be construed as a generalization of anyinventive subject matter disclosed herein and shall not be considered tobe an element or limitation of the appended claims except whereexplicitly recited in a claim(s).

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Embodiments of the disclosure may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentdisclosure, a user may access crowd sourcing applications or relateddata available in the cloud. For example, the data exception emittercould execute on a computing system in the cloud and emit data qualityexceptions. In such a case, the crowd sourcing manager could receive theexception and store details related to the exception at a storagelocation in the cloud. Doing so allows a user to access this informationfrom any computing system attached to a network connected to the cloud(e.g., the Internet).

It is understood in advance that although this disclosure includes adetailed description on cloud computing, implementation of the teachingsrecited herein are not limited to a cloud computing environment. Rather,embodiments of the present disclosure are capable of being implementedin conjunction with any other type of computing environment now known orlater developed.

For convenience, the Detailed Description includes the followingdefinitions which have been derived from the “Draft NIST WorkingDefinition of Cloud Computing” by Peter Mell and Tim Grance, dated Oct.7, 2009, which is cited in an IDS filed herewith, and a copy of which isattached thereto.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g. networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure comprising anetwork of interconnected nodes. A node in a cloud computing network isa computing device, including, but not limited to, personal computersystems, server computer systems, thin clients, thick clients, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputer systems, mainframe computer systems, and distributed cloudcomputing environments that include any of the above systems or devices,and the like. A cloud computing node is capable of being implementedand/or performing any of the functionality set forth hereinabove.

FIG. 1A is a block diagram illustrating a schematic 100 to transparentlyenforce policies in distributed processing infrastructures, according toone embodiment disclosed herein. The schematic 100 depicts a distributedprocessing infrastructure, which, in one embodiment, is a Hadoopinfrastructure. A name node 101 orchestrates the overall infrastructure.The name node 101 is responsible for receiving tasks from externalconsumers 150 _(1-N) and delegating the tasks to at least one of aplurality of data nodes, which in the schematic 100, include data nodes111-128. The information stored on the name node 101 is encrypted. Theconsumers 150 _(1-N) are the systems using the service provided by theinfrastructure orchestrated through the name node 101 by subscribing tothe service. The consumers 150 _(1-N), when subscribing to the service,provide a service level agreement (SLA) and associated policies.

The name node 101 also includes a repository 110, which is configured tohold at least two distinct types of data—data node metadata and policymetadata. Data node metadata describes the characteristics of each datanode in the infrastructure, which may include, but is not limited to:(i) security related attributes, such as privacy settings, encryptionsettings, retention settings, etc., (ii) technical attributes, such asCPU, memory, and storage capabilities, (iii) geographical information,such as physical location (address), and any applicable laws andregulations for the country or region (e.g., European Union), (iv)organizational information such as corporation, business unit, owner ofthe hardware infrastructure (e.g., in private cloud environments,computing equipment might be funded by different business unites, andsharing of it across major business unit boundaries may or may not bepermissible), (v) administrator contact details, (vi) data node resourceutilization, and (vii) use case metadata, such as whether the data nodemay be used for disaster recovery use cases. In some embodiments, thename node may retrieve the data node metadata, or an administrator mayenter the data node metadata.

Policies (and their metadata) stored in the repository 110 are providedby consumers. Once a subscription for the service is made, the name nodemust persist the policy (or policies) with additional information on howthe SLA and policies are deployed. The name node must therefore knowwhich data nodes have been selected to fulfill the service subscriptionand why they have been selected. The policies may specify differentrequirements, including, but not limited to: (i) a data volumerequirement, (ii) security and privacy aspects of the data (for example,it may be permissible to have transactional business object data likemarketing campaign information on any data node, whereas master databusiness objects like sales employee information might only be permittedon data nodes fulfilling certain security and privacy requirements),(iii) redundancy requirements, (iv) disaster recovery (e.g., minimaldistance of data nodes for scope of different types of disasters), and(v) legal requirements (e.g., certain regulations demand that data doesnot cross country borders, therefore requiring knowledge of where thedata nodes are physically located). In some embodiments, the policy maybe received with the service request, but may not need to be stored inthe repository 110. For example, the policy related to a one-timeservice request may or may not be stored in the repository 110.

As shown, a plurality of secondary name nodes 102 _(1-M) provideredundancy and disaster recovery solutions for the name node 101, whichis a single point of failure for the overall infrastructure. There maybe one or multiple secondary name nodes 102 _(1-M) depending on theparticular requirements of the deployment. The secondary name nodes 102_(1-M) communicate with the name node 101 over a secure connection.

The data nodes 111-128 are used for algorithmic and data processing. Asshown, the data nodes are grouped into three different types, namelygroups 106-108. The data nodes may be grouped based on any number ofattributes, including but not limited to geography, organization,hardware attributes. In any implementation there may be any number ofdifferent data node types, with any number of each type of data node.Additionally, the data nodes may be configured so that the individualdata nodes do not contain information identifying the other data nodesin the cloud. Such an embodiment may be advantageous, for instance, incase a data node is hacked by a malicious party. In such a situation, itwould not be possible for the malicious party to use metadata stored onthe hacked data node to find out which other data nodes belong to thesame cloud.

As shown, two example datasets 103 and 104 have been sent to the namenode 101 for processing pursuant to a subscription and the attachedpolicies provided by one of the consumers 150 _(1-N). Once received, thename node 101 must determine whether one (or more) of the data nodes111-128 are suitable for handling the processing and storage of thedatasets 103-104. In making this determination, the name node 101 readsthe data node metadata from the repository 110, parses the policies, andretrieves data node utilization figures. The name node 101 thendetermines whether current (and future) data node utilization allows forsufficient free capacity to accommodate the new workload. If there isnot sufficient capacity, the name node 101 rejects the new workload. Ifsufficient capacity exists, the name node 101 determines whether thepolicies can be fulfilled considering the details in an implementationstrategy it devises. If the policies cannot be fulfilled, the name node101 rejects the new workload. If the policies can be fulfilled, the namenode 101 computes the physical layout of the workload on the data nodes111-128 based on hardware characteristics of the data nodes, performanceattributes, availability of the data nodes, and disaster recoveryrequirements. In some embodiments, the name node 101 may assign asuitability score for each data node 111-128, which incorporates theability to fulfill policy requirements and the above-mentionedattributes of the data nodes. If a possible layout is determined, thename node 101 accepts the workloads of datasets 103-104.

FIG. 1B is a block diagram illustrating the deployed workload ofdatasets 103-104 after acceptance by the name node 101. As shown, thename node 101 has partitioned the datasets 103-104 into three distinctportions each for processing. Data node 111 has portion 103 ₁, data node112 has portion 103 ₂, and data node 114 has portion 103 ₃. As shown,each portion of dataset 103 has been deployed to a data node in group106. The name node 101 may have made this determination on any number offactors, based on the metadata of each data node and the policyrequirements in the repository 101. The nodes of the group 106 may havebeen determined because they are located in a state (e.g., NorthCarolina) in which the policy mandates the dataset 103 must reside. Thenodes 111, 112, and 114 may have been selected because they had lowerresource utilizations (and therefore higher suitability scores) than theother nodes. The free blocks on the compute nodes indicate that the datanodes may be able to process data for other subscribers (and maycurrently be processing such data).

As shown, each portion of dataset 104 has been deployed to data nodes ingroup 107. Data node 119 has portion 104 ₁, data node 121 has portion104 ₂, and data node 122 has portion 104 ₃. Each portion of dataset 103has been deployed to a data node in group 107. The name node 101 mayhave made this determination on any number of factors, based on themetadata of each data node and the policy requirements in the repository101. For example, the dataset 104 might have sensitive data which mayonly be protected by the nodes of group 107. Alternatively, the nodes ofgroup 107 may have been chosen by the name node 101 because they areowned by the consumer owning the dataset 104, and the policy mandatesthat the dataset 104 be processed and stored on the consumer'sresources. The nodes 119, 121, and 122 may have been selected becausethey had lower resource utilizations (and therefore higher suitabilityscores) than the other nodes. It should be noted that although each ofportion the datasets 103 and 104 were deployed to the same group, underappropriate circumstances, the portions may be distributed among datanodes of different groups.

Upon accepting and deploying the workload, the name node 101 thenupdates the metadata in the repository 110 regarding the processingtasks on each data node receiving new workload. The configurationdepicted FIG. 1B is the result of the name node 101 creating a dynamiccloud infrastructure with placement of data onto data nodes which arecompliant with all relevant policies. The data nodes selected are alsothe best suited for the workload in light of resource utilization of thedata nodes. Therefore, embodiments disclosed herein provide aninfrastructure where the data nodes are not equal due to their varyingattributes, as discussed above. The metadata corresponding to theseattributes is utilized by the name node 101 to make decisions on whetherto accept workloads, and how to deploy the workloads.

FIG. 2 is a block diagram illustrating a system 200 to transparentlyenforce policies in distributed processing infrastructures. The system200 includes a computer 202. In one embodiment, the computer 202 is thename node 101 of FIG. 1. The computer 202 may also be connected to othercomputers via a network 230. The computer 202 may also be connected toseveral backup computers for failover and disaster recovery purposes(not shown). In general, the network 230 may be a telecommunicationsnetwork and/or a wide area network (WAN). In a particular embodiment,the network 230 is the Internet.

The computer 202 generally includes a processor 204 connected via a bus220 to a memory 206, a network interface device 218, a storage 208, aninput device 222, and an output device 224. The computer 202 isgenerally under the control of an operating system (not shown). Examplesof operating systems include the UNIX operating system, versions of theMicrosoft Windows operating system, and distributions of the Linuxoperating system. (UNIX is a registered trademark of The Open Group inthe United States and other countries. Microsoft and Windows aretrademarks of Microsoft Corporation in the United States, othercountries, or both. Linux is a registered trademark of Linus Torvalds inthe United States, other countries, or both.) More generally, anyoperating system supporting the functions disclosed herein may be used.The processor 204 is included to be representative of a single CPU,multiple CPUs, a single CPU having multiple processing cores, and thelike. Similarly, the memory 206 may be a random access memory. While thememory 206 is shown as a single identity, it should be understood thatthe memory 206 may comprise a plurality of modules, and that the memory206 may exist at multiple levels, from high speed registers and cachesto lower speed but larger DRAM chips. The network interface device 218may be any type of network communications device allowing the computer202 to communicate with other computers via the network 230.

The storage 208 may be a persistent storage device. Although the storage208 is shown as a single unit, the storage 208 may be a combination offixed and/or removable storage devices, such as fixed disc drives, solidstate drives, floppy disc drives, tape drives, removable memory cards oroptical storage. The memory 206 and the storage 208 may be part of onevirtual address space spanning multiple primary and secondary storagedevices.

As shown, the memory 206 contains the delegation manager 212, which isan application generally configured to perform the functionality of thename node 101 described in FIG. 1. The delegation manager 212 storesmetadata regarding a plurality of data nodes 240 _(1-N) in therepository 210. The metadata of the data nodes 240 _(1-N) describescharacteristics of the data nodes 240 ₁-N, including, but not limitedto: (i) security related attributes, such as privacy settings,encryption settings, retention settings, etc., (ii) technicalattributes, such as CPU, memory, and storage capabilities, (iii)geographical information, such as physical location (address),applicable laws and regulations for each country or region (e.g.,European Union), (iv) organizational information such as corporation,business unit, owner of the hardware infrastructure (e.g., in privatecloud environments, computing equipment might be funded by differentbusiness unites, and sharing of it across major business unit boundariesmay or may not be permissible), (v) administrator contact details, (vi)data node resource utilization, and (vii) use case metadata, such aswhether the data node may be used for disaster recovery use cases.

The delegation manager 212 may receive service requests from a pluralityof consumers 250 _(1-N). The requests include service level agreements(SLA) and policies, which are stored in the repository 210. The policiesmay specify different requirements, including, but not limited to: (i) adata volume requirement, (ii) security and privacy aspects of the data(for example, it may be permissible to have transactional businessobject data like marketing campaign information on any data node,whereas master data business objects like sales employee informationmight only be permitted on data nodes fulfilling certain security andprivacy requirements), (iii) redundancy requirements, (iv) disasterrecovery (e.g., minimal distance of data nodes for scope of differenttypes of disasters), and (v) legal requirements (e.g., certainregulations demand that data does not cross country borders, thereforerequiring knowledge of where the data nodes are physically located).Upon receiving a service request, the delegation manager 212 makes adetermination as to whether the request can be fulfilled in light of thecurrent workload on the data nodes 240 _(1-N) by retrieving metadatarelated to the workload capacity of each data node 240 _(1-N). Ifsufficient capacity exists, the delegation manager 212 determineswhether the request's policies and SLA can be fulfilled in light of themetadata of each data node 240 ₁-N. If the policies and SLA can befulfilled, the delegation manager 212 may partition the workload intodiscrete tasks for deployment to a select number of data nodes 240_(1-N), where they are processed and/or stored.

As shown, the memory 206 also contains the cloud admin UI 214. The cloudadmin UI 214 is an application which provides a user interface foradministrators configuring and maintaining the infrastructure operatedthrough the name node (computer 202).

As shown, the storage 208 contains the repository 210, which storesmetadata related to data nodes 240 _(1-N), metadata related to consumerpolicies, and service level agreements. Although depicted as a database,the repository 210 may take any form sufficient to store data, includingtext files, xml data files, and the like. Although depicted as part ofthe computer 202, the repository 210 may be stored in a system externalto the computer 202.

The input device 222 may be any device for providing input to thecomputer 202. For example, a keyboard and/or a mouse may be used. Theoutput device 224 may be any device for providing output to a user ofthe computer 202. For example, the output device 224 may be anyconventional display screen or set of speakers. Although shownseparately from the input device 222, the output device 224 and inputdevice 222 may be combined. For example, a display screen with anintegrated touch-screen may be used.

FIG. 3 is a flow chart illustrating a method 300 to transparentlyenforce policies in distributed processing infrastructures, according toone embodiment disclosed herein. In one embodiment, the distributingprocessing infrastructure is a Hadoop processing infrastructure having aname node and a plurality of data nodes, as described above, and thesteps of the method 300 may be performed by the delegation manager 212.At step 310, the delegation manager 212 provides and manages data nodemetadata at the name node. Described in greater detail above, the datanode metadata generally describes characteristics of each data node inthe computing infrastructure. At step 320, the delegation manager 212receives a service request from a consumer. Exemplary services providedinclude, but are not limited to, data analysis tasks such as dataprofiling, data aggregation, data transformation, search for duplicaterecords, and search for a compound unique key. Any type of analysiswhich can be executed in parallel in a distributed environment may beprovided. The service request may generally include a service levelagreement and policies specifying a number of different requirements,which may relate to data volume requirements, security/privacyrequirements, redundancy requirements for affected service availability,disaster recovery requirements, and legal requirements. At step 330, thedelegation manager 212 may store the policy metadata in the repository210, and may parse the policy to determine the consumer's exactrequirements in light of the request. At step 340, the delegationmanager 212 may identify data nodes suitable to accept the requesteddata processing workload. The delegation manager 212 may identifysuitable data nodes based on data node metadata and the policy metadata.At step 350, if the delegation manager 212 identifies suitable datanodes, the workload is deployed.

FIG. 4 is a flow chart illustrating a method 400 corresponding to step340 for identifying data nodes suitable to accept a data processingworkload, according to one embodiment disclosed herein. In someembodiments, the delegation manager 212 performs the steps of the method400. At step 400, the delegation manager 212 retrieves the data nodeutilization metadata for each data node in the computing infrastructure.In some embodiments, the data node utilization metadata is stored in therepository 210. The data node utilization metadata may containinformation including, but not limited to, current (and scheduled)workload for the data node, including a number of tasks to be completed,current resource utilization (such as CPU, memory, and storage capacity)of the data node, and expected resource utilization in completingscheduled tasks.

At step 420, the delegation manager begins executing a loop includingsteps 430-460 for each data node in the computing infrastructure. Atstep 430, the delegation manager 212 determines, based on the data nodeutilization metadata, whether the data node has sufficient free capacityto accept at least a portion of the workload. In some embodiments, thedelegation manager 212 may compute a utilization score based on thecurrent workload of the data node to determine a utilization level. Thedelegation manager 212 may then compare the utilization score to apredefined maximum utilization threshold. If the utilization score islower than the threshold, the data node is considered as havingsufficient free capacity to accept a new workload. If the data node doesnot have sufficient free capacity, the delegation manager 212 proceedsto step 460. If the data node has sufficient free capacity, thedelegation manager 212 proceeds to step 440, where it determines whetherthe data node can fulfill each aspect of the consumer's policy. Forexample, a consumer located in the European Union may require thatpersonal information must be protected from loss, unauthorizeddisclosure, and modification while the data is in transit or at rest.Therefore, the delegation manager 212 may determine whether the datanode can meet these requirements. If the data node cannot fulfill theserequirements, the delegation manager 212 proceeds to step 460.Otherwise, the delegation manager 212 proceeds to step 450. At step 450,the delegation manager 212 computes a suitability score for the datanode and adds the data node to a list of candidate data nodes. Thesuitability score may be based on the metadata attributes of the datanode, ability to fulfill policy requirements, and available capacity.Any suitable scoring range and any method for assigning a score may beimplemented. At step 460, the delegation manager 212 determines whethermore data nodes remain to be scored. If more data nodes remain, thedelegation manager 212 returns to step 420. Otherwise, the delegationmanager 212 proceeds to step 470.

At step 470, the delegation manager 212 computes the physical layout ofthe workload to determine whether a possible layout to distribute theworkload exists. At this step, the delegation manager 212 may considereach data node on the candidate list, and assign portions of theworkload to the data nodes having the highest suitability scores. Forexample, the data node with the highest suitability score may beselected over the data node with the second highest suitability score.If additional data nodes need to be selected because the workload cannotbe accommodated by a single data node, the data node with the secondhighest suitability score may be selected, although another node havinga sufficient suitability score may be selected. At step 480, if apossible layout is found, the delegation manager 212 accepts theworkload and updates the data node metadata to reflect the newlyassigned workloads.

In the event that an attribute of a data node is not populated in thename node metadata, the delegation manager 212 may take differentcourses of action based on the nature of the attribute. If the attributeis security related, the data node is marked as not fulfilling thesecurity requirement and is excluded from the candidate list of datanodes (or not added to the list to begin with) to avoid potentialsecurity issues. For example, if the cloud administrator does not wantto specify all attributes for all data nodes, this approach prevents thesituation where a data node is added with manipulated metadata. If theattribute is not security related, however, the data node might beselected if there is no other data node available which explicitlyfulfils the requirement.

Additionally, the consumer policies can be variable, following aworkflow approach. For example, the policy may be constant over time,less strict on subsequent processing steps, or stricter on subsequentprocessing steps. In these scenarios, the name node can build the cloudfor each processing step based on the policy applicable for eachprocessing step. If there are advanced security requirements, the namenode might enforce a policy that a data node with sensitive data mayonly be part of one cloud.

Advantageously, embodiments provide a dynamic orchestration of cloudcomputing infrastructure through the name node of the infrastructure.Additionally, embodiments provide support for consumer policies,providing solutions for different requirements, such as: (i) dataprivacy, (ii) data security, (iii) disaster recovery, (iv) performance,and (v) infrastructure sharing. Embodiments also provide autonomicfeatures in the HDFS infrastructure for policy support.

Referring now to FIG. 5, illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 4 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 6, a set of functional abstraction layers providedby cloud computing environment 50 (FIG. 5) is shown. It should beunderstood in advance that the components, layers, and functions shownin FIG. 5 are intended to be illustrative only and embodiments of thedisclosure are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include mainframes, in oneexample IBM® zSeries® systems; RISC (Reduced Instruction Set Computer)architecture based servers, in one example IBM pSeries® systems; IBMxSeries® systems; IBM BladeCenter® systems; storage devices; networksand networking components. Examples of software components includenetwork application server software, in one example IBM WebSphere®application server software; and database software, in one example IBMDB2® database software. (IBM, zSeries, pSeries, xSeries, BladeCenter,WebSphere, and DB2 are trademarks of International Business MachinesCorporation registered in many jurisdictions worldwide.).

Virtualization layer 62 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer 64 may provide the functions describedbelow. Resource provisioning provides dynamic procurement of computingresources and other resources that are utilized to perform tasks withinthe cloud computing environment. Metering and Pricing provide costtracking as resources are utilized within the cloud computingenvironment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal provides access to the cloud computing environment forconsumers and system administrators. Service level management providescloud computing resource allocation and management such that requiredservice levels are met. Service Level Agreement (SLA) planning andfulfillment provide pre-arrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

Workloads layer 66 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and policy enforcement in distributed processinginfrastructures.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A computer program product comprising: anon-transitory computer-readable storage medium having computer-readableprogram code embodied therewith, the computer-readable program codeexecutable by a processor to perform an operation comprising: receiving,by a name node, a request to process a client workload on a subset of aplurality of data nodes in a distributed computing environment, whereinthe name node stores a file system index reflecting files stored on theplurality of data nodes as part of a distributed file system of thedistributed computing environment; retrieving, by the name node, a setof requirements for processing the client workload; analyzing, by thename node, the retrieved set of requirements and nodal data describingeach of the data nodes, to select, a first data node of the plurality ofdata nodes as a delegation target to process at least a portion of theclient workload, the first data node being selected upon determining:(i) the first data node has a level of resource utilization notexceeding a maximum delegation threshold, and (ii) the nodal data of thefirst data node satisfies a greater count of the set of requirementsthan the nodal data of a second data node of the plurality of datanodes, wherein the set of requirements and the nodal data are encryptedand stored on the name node; delegating the requested processing of theclient workload to the delegation target; and updating the nodal data toreflect the delegation of the client workload to the delegation target,wherein each of the data nodes does not include nodal data identifyingother data nodes.
 2. The computer program product of claim 1, whereinthe set of requirements is specified by a user in a service levelagreement and comprises: (i) a data volume requirement, (ii) a securityrequirement, (iii) a privacy requirement, (iv) a redundancy requirement,(v) a disaster recovery requirement, and (vi) a legal requirement. 3.The computer program product of claim 2, wherein the nodal datacomprises: (i) a data node security attribute, (ii) a data node hardwareattribute, (iii) a geographical location of the data node, (iv) a datanode organizational information attribute, (v) a level of utilization ofone or more resources of the data node, and (vi) a data node disasterrecovery attribute.
 4. The computer program product of claim 3, theoperation further comprising computing a suitability score for each datanode, wherein the suitability score for a given data node is higher ifthe nodal data of the given data node allows the given data node tosatisfy a greater count of the set of requirements, wherein the firstdata node is selected as the delegation target upon determining thesuitability score of the first data node is greater than the suitabilityscore of the second data node.
 5. The computer program product of claim4, wherein the name node is not configured to be a delegation target,wherein each of the data nodes is not configured to retrieve the nodaldata of any other data node.
 6. The computer program product of claim 5,wherein the set of requirements is at least partially stored in a datapolicy comprising: (i) a data retention policy, (ii) a data securitypolicy, (iii) a data privacy policy, (iv) a performance policy, (v) adisaster recovery policy, and (vi) an infrastructure sharing policy. 7.A system, comprising: one or more computer processors; and a memorycontaining a program which, when executed, performs an operationcomprising: receiving, by a name node, a request to process a clientworkload on a subset of a plurality of data nodes in a distributedcomputing environment, wherein the name node stores a file system indexreflecting files stored on the plurality of data nodes as part of adistributed file system of the distributed computing environment;retrieving, by the name node, a set of requirements for processing theclient workload; analyzing, by the name node, the retrieved set ofrequirements and nodal data describing each of the data nodes, toselect, a first data node of the plurality of data nodes as a delegationtarget to process at least a portion of the client workload, the firstdata node being selected upon determining: (i) the first data node has alevel of resource utilization not exceeding a maximum delegationthreshold, and (ii) the nodal data of the first data node satisfies agreater count of the set of requirements than the nodal data of a seconddata node of the plurality of data nodes, wherein the set ofrequirements and the nodal data are encrypted and stored on the namenode; delegating the requested processing of the client workload to thedelegation target; and updating the nodal data to reflect the delegationof the client workload to the delegation target, wherein each of thedata nodes does not include nodal data identifying other data nodes. 8.The system of claim 7, wherein the set of requirements is specified by auser in a service level agreement and comprises: (i) a data volumerequirement, (ii) a security requirement, (iii) a privacy requirement,(iv) a redundancy requirement, (v) a disaster recovery requirement, and(vi) a legal requirement.
 9. The system of claim 8, wherein the nodaldata comprises: (i) a data node security attribute, (ii) a data nodehardware attribute, (iii) a geographical location of the data node, (iv)a data node organizational information attribute, (v) a level ofutilization of one or more resources of the data node, and (vi) a datanode disaster recovery attribute.
 10. The system of claim 9, theoperation further comprising computing a suitability score for each datanode, wherein the suitability score for a given data node is higher ifthe nodal data of the given data node allows the given data node tosatisfy a greater count of the set of requirements, wherein the firstdata node is selected as the delegation target upon determining thesuitability score of the first data node is greater than the suitabilityscore of the second data node.
 11. The system of claim 10, wherein thename node is not configured to be a delegation target, wherein each ofthe data nodes is not configured to retrieve the nodal data of any otherdata node.
 12. The system of claim 11, wherein the set of requirementsis at least partially stored in a data policy comprising: (i) a dataretention policy, (ii) a data security policy, (iii) a data privacypolicy, (iv) a performance policy, (v) a disaster recovery policy, and(vi) an infrastructure sharing policy.
 13. A computer program productcomprising: a non-transitory computer-readable storage medium havingcomputer-readable program code embodied therewith, the computer-readableprogram code executable by a processor to perform an operationcomprising: receiving, by a name node, a request to process a clientworkload on a subset of a plurality of data nodes in a distributedcomputing environment; retrieving, by the name node from a service levelagreement, a set of requirements for processing the client workload,wherein the set of requirements comprise: (i) a data volume requirement,(ii) a security requirement, (iii) a privacy requirement, (iv) aredundancy requirement, (v) a disaster recovery requirement, and (vi) alegal requirement; computing, by the name node, a suitability score foreach data node, wherein the suitability score for each data node isbased on a count of the set of requirements satisfied by the respectivenodal data of each data node, wherein the nodal data for each data nodecomprises: (i) a data node security attribute, (ii) a data node hardwareattribute, (iii) a geographical location of the data node, (iv) a datanode organizational information attribute, (v) a level of utilization ofone or more resources of the respective data node, and (vi) a data nodedisaster recovery attribute; and selecting, by the name node, a firstdata node of the plurality of data nodes as a delegation target toprocess at least a portion of the client workload upon determining thesuitability score of the first data node is greater than the suitabilityscore of a second data node of the plurality of data nodes, wherein theset of requirements and the nodal data are encrypted and stored on thename node.
 14. The computer program product of claim 13, wherein aplurality of secondary name nodes provide redundancy and disasterrecovery for the name node, wherein each of the data nodes does notinclude data identifying other data nodes, the operation furthercomprising: delegating the requested processing of the client workloadto the delegation target; and updating the nodal data to reflect thedelegation of the client workload to the delegation target.
 15. Thecomputer program product of claim 14, wherein the first data node isselected upon determining that the first data node has a level ofresource utilization not exceeding a maximum delegation threshold,wherein the name node stores a file system index reflecting files storedon the plurality of data nodes as part of a distributed file system ofthe distributed computing environment.